Time-of-flight (TOF) systems that provide a measure of distance (Z) from the system to a target object without depending upon luminosity or brightness information obtained from the target object are known in the art. See for example U.S. Pat. No. 6,323,942 entitled CMOS-Compatible Three-Dimensional Image Sensor IC (2001), assigned to Canesta, Inc., now of Sunnyvale, Calif. TOF systems according to the '942 patent include a light source of relatively low power periodic optical energy typically in the near IR (NIR) range of perhaps 800 nm) and determine how long it takes until at least some of that energy reflected by a target object arrives back at the system to be detected. Detection occurs within a typically 50 μm×50 μm array of Z-pixel detectors, perhaps 100×100 pixels, as well as circuitry for processing detection charge output by the associated detector, e.g., 100×100 processing circuits. Some of emitted optical energy (e.g., from the light source) can reflect off the surface of a target object and is detected by pixels in the pixel array. The roundtrip length of time or time-of-flight (TOF) between emitted optical energy and detection of reflected optical energy is a measure of the separation distance z between the TOF system and the target object. Thus, if the roundtrip TOF time is denoted t1, then the distance between target object and the TOF system is Z1, where Z1=t1·C/2, and where C is velocity of light. The TOF system may also include a microprocessor or microcontroller unit with associated RAM and ROM, as well as a high speed distributable clock and various computing and input/output (I/O) circuitry. Among other functions, the controller unit can perform z-distance to target object and target object velocity calculations. TOF systems can function well to determine z-distance to a target object and can realize three-dimensional images of a target object in real time, without need for ambient light.
More sophisticated TOF approaches are exemplified by U.S. Pat. Nos. 6,515,740 (2003) and 6,580,496 (2003) respectively Methods and Systems for CMOS-Compatible Three-Dimensional Imaging Sensing Using Quantum Efficiency Modulation, assigned to Canesta, Inc., now of Sunnyvale, Calif. In these systems, TOF is determined by examining relative phase shift between transmitted light signals and light signals reflected from the target object. Detection of the reflected light signals over multiple locations in the system pixel array results in measurement signals that are referred to as depth or z-images. The depth images represent a three-dimensional image of the target object surface.
Conventional RGB systems are known in the art and can include such commonly used devices as color cameras, home video cameras, and the like. Such systems require ambient light, e.g., sunlight, to illuminate a target object. Ambient optical energy reflected by the target object falls upon an array of red-green-blue (RGB) sensitive pixels in an array. Assuming sufficient ambient light, the array of RGB pixels can, more or less, develop an image representative of what the camera is seeing.
Thus, while TOF systems acquire z-data, RGB systems essentially acquire a color or gray-scale, or black-and-white (BW), image of the target object. In some applications it is useful to simultaneously acquire both RGB and z information.
FIG. 1A depicts a generic camera or imaging system 10 that includes a preferably RGB-Z sensor 20 containing pixels responsive to optical energy of RGB wavelengths about 400 nm to about 650 nm and containing other pixels sensitive to IR or NIR wavelengths (about 700 nm to about 900 nm). System 10 in FIG. 1A (and in FIG. 1B) is similar to an embodiment described in U.S. patent application Ser. No. 11/044,996 (published as US 2005/0285966) entitled Single Chip RGB-Z sensor, assigned to Canesta, Inc., assignee herein. As described below, the configuration of FIG. 1A does not provide a common global optical filtering path for optical energy falling upon the RGB pixels and the Z pixels due to the presence of a beam splitting mechanism 70. Stated differently, optical energy falling upon RGB pixels has different wavelengths than optical energy falling upon Z pixels, because the two types of wavelengths have already been discriminated at a global level by the beam splitting mechanism 70. But as a practical matter, in some applications it may simply not be feasible to implement beam splitting.
In the beam splitting embodiment of FIG. 1A, optical energy to be detected by system 10 passes through a focusing lens 15. In the embodiment shown, RGB-Z 20 includes a first substrate 30 containing array 40 of high resolution RGB pixel detectors, and a second substrate 50 containing array 60 of lower resolution Z-pixel detectors responsive to IR or NIR wavelengths. RGB pixel array 40 outputs signals proportional to detected RGB optical energy, about 400 nm to about 650 nm. Generally IR or NIR Z-pixel array 60 is responsive to wavelengths of about 700 nm to about 900 nm. An optically selective material 70, for example a hot mirror, permits RGB optical energy to fall upon RGB array 40, while allowing IR or NIR optical energy to fall upon Z-array 60. Thus, two different optical paths are provided for optical energy exiting the beam splitting mirror mechanism 70: one path (upward in FIG. 1A) for IR-NIR wavelengths, and a second path (leftward in FIG. 1A) for RGB wavelengths. Generally if optical energy falling upon structure 70 has wavelengths exceeding perhaps 700 nm, the energy is IR-NIR and is reflected (upward in FIG. 1A) for detection by IR-NIR pixel array 50. However if optical energy falling upon material 70 has wavelength less than about 700 nm, the optical energy is RGB and passes through structure 70 for detection by RGB pixel array 30.
Detection of RGB optical energy relies upon the existence of a source of ambient light 80, the sun perhaps. Some RGB energy 90 falls upon target object 40 and is reflected into system 10 to be detected by RGB pixel sensor array 40. An RGB processor 110 processes the sensor 40 output and can cause an RGB or BW image of the target object to be present on a display. Understandably, if ambient light source 80 outputs too little or too much optical energy, RGB array 40 may not provide any meaningful information.
Detection of NIR or IR energy is a bit more complicated. Typically a modulator 130 under control of a clock system and microprocessor (not shown for ease of illustration) causes a source of NIR or IR optical energy 130 to emit period modulations. Some of this NIR or IR energy reflects off target object 40 and is detected by z-pixel array 60. A z-processor 150 (typically under control of a clock system and microprocessor, not shown) receives output signals from array 60 and derives TOF information, including z-distance data.
FIG. 1B depicts another embodiment of an imaging system 10 that also includes a beam splitting mechanism 70-1, but provides provides RGB and Z pixel detectors in a common plane, e.g., IC substrate 160. Substrate 160 includes a singe array structure 170 having regions with an array of RGB pixels 180 and regions with an array of z-pixels 190. The configuration of FIG. 1B, like that of FIG. 1A, does not provide a common global optical path for optical energy reaching RGB pixels and the Z pixels. The RGB optical system 20-1 includes a beam splitter 70-1, for example a hot mirror. In the configuration shown, hot mirror 70-1 substantially passes RGB wavelengths for detection by RGB sensor array 180, while deflecting (upward in FIG. 1B) IR-NIR components. In the IR-NIR optical system 20-2, a mirror 70-2 deflects incoming IR-NIR optical energy through an optical path equalizer 200 to Z-array 190, for detection. Other components in FIG. 1B may be substantially identical to what was described or alluded to with respect to FIG. 1A.
But the presence of beam splitting mechanism 70-1 can restrict the desired configuration of RGB and Z pixels. For example, FIG. 1C depicts an exemplary layout for single array 160, depicting R pixels, G pixels, B pixels, and Z-pixels (denoted NIR) that might not readily work with the configuration of FIG. 1B unless perhaps the RGB pixels and the z pixels were each grouped in substantially contiguous regions. Yet in some applications, it may be desired to implement an RGB-Z imaging system with precisely the sort of array shown in FIG. 1C.
Thus, there is a need in an RGB-Z imaging system to provide a global optical path. Among other advantages, such a configuration would enable great flexibility in interspersing RGB pixels and Z pixels on a common substrate, e.g., see FIG. 1C. (Note, while FIG. 1C shows IR-NIR pixels physically larger than R, G, or B pixels, such need not be the case.) As such, R pixels, G pixels, B pixels and Z pixels would all be subject to receiving optical energy with the same spectral content but for the presence of some sort of pre-pixel filter. But providing filters on a per pixel basis commonly requires using so-called dye filters (sometimes called bulk filters). But dye filters cannot readily be implemented with narrowly defined pass bands. A typical pass band for a dye filter might be about 100 nm or greater, with typically poor frequency roll-off characteristics. But the sloppy characteristics associated with bulk filters does not lend such filters to use with Z pixels, which pixels require a very narrow bandwidth, e.g., about 30 nm.
It is known in the art to fabricate narrow band filters such as interference or dichotic filters for use with optical energy of a relatively narrow bandwidth. Thus within a critical region bandwidth of 700 nm to 900 nm, an interference filter with a pass band of 800 nm to 830 nm may transmit 90% of optical energy within the pass band, while blocking perhaps 99% of optical energy whose wavelengths are beyond the filter pass band. But the specification for such filters is typically silent as to the filter performance in a non critical region, perhaps the 400 nm to 700 nm range. Filter designers do not specify such performance because the narrowband filter is typically coupled with an inexpensive wideband filter that attenuates optical energy in the non critical region of the interference filter, perhaps from 400 nm to 750 nm. By coupling both filters together, a sharp filter with a pass band of 800 nm to 830 nm and that attenuates optical energy elsewhere over the 400 nm to 900 nm range is achieved. But as the filter specification is generally silent as to transmission characteristics in the non critical 400 nm to 700 nm range, such filters typically pass optical energy in that wavelength band and are not suitable for use with systems such as shown in FIGS. 1A and 1B.
Thus there is a need for an RGB-Z imaging system that provides a global optical path. Further there is a need to provide such imaging system with preferably per pixel dye filters such that R, G, B, and Z pixels each receive substantially only the wavelengths intended to be detected by these pixels. Thus, each R, each G, each B, and each Z pixel preferably would have a dedicated dye filter. But as noted, dye filters are characterized by sloppy bandwidths. Thus, there is a need to provide a filter mechanism to ensure that optical energy presented to the Z pixels is very narrow in bandwidth, e.g., about 30 nm.
In addition to providing improved pixel filter discrimination, systems such as shown in FIGS. 1A and 1B can What is also needed is a method and system by which the relative vantage point of the system can synthetically be made to move, as though a human observer were examining the target object or scene through a movable mirror, to enhance knowledge of the target object.
The present invention provides such methods and systems to inexpensively implement RGB-Z imaging systems having a global optical path, with per pixel dye filters yet providing sharp bandwidth filtering for Z-pixels, as well as to enhance imaging by use of a synthetic mirror.